Automating recurrent data science tasks remains one of the most promising areas of artificial intelligence. Agents built on large language models can clean data, write SQL queries, select statistical tests, and format results. However, a practical question arises: is it enough to provide the model with a predefined set of instructions, or can those instructions even make the outcome worse?
In this context, what are known as LLM-generated skills are presented as a way to avoid starting every execution from scratch. The idea is simple: a model writes a reusable file with guidance for a family of tasks, and data teams then integrate it into their workflows. In theory, that removes the bottleneck of writing and maintaining instructions manually. But recent empirical evidence does not support that confidence.
A study with a rigorous design compared the performance of a complete LLM-generated skill against using no skill at all. The research covered four phases of the data science lifecycle: preparation, extraction, statistical analysis, and reporting. In the main analysis, 56 tasks, nine model configurations, and three providers were executed, adding up to 7,560 runs. The result was clear: neither the complete skill nor any trimmed variant significantly improved performance. All p-values were at least 0.396 and the total spread across variants was only 1.2 percentage points.
A second experiment, designed to match token usage, added 1,512 runs. In that test, complete skills behaved similarly to task-irrelevant skill-formatted content. In other words, simply adding an automatically generated template does not provide useful information to the model. It can even distract attention or introduce conventions that do not apply to the specific problem.
This finding does not invalidate the concept of skills in general. Guidance written by an expert, tested and adjusted, can be very valuable. What it questions is the idea that a skill generated without curation can become the default strategy. For companies, the lesson is important: it is not enough to ask an LLM to write instructions; you have to design the entire system around the model.
At Q2BSTUDIO, a software and technology development company, we take a different approach. Instead of relying on generic skill files, we recommend building custom software that incorporates validated business logic. A well-designed application can run AI agents, control access to data, and evaluate each response automatically. Thus, the prompt is just one piece of a robust mechanism, not the only defense.
In fact, the development of AI agents must go beyond templates. An agent in production decides when to use a skill, which examples to include, and when to ask a human. That level of orchestration is not achieved with a static file, but with a software architecture prepared to observe, measure, and correct the model's behavior. This is where software engineering experience makes the difference.
Infrastructure matters too. Data science pipelines often run on cloud AWS/Azure, where storage, compute, and governance must be integrated. Custom software allows leveraging those services without being tied to a closed tool. It also facilitates traceability of each experiment and deployment of models with version control. In real projects, the cloud is not a complement; it is the operational foundation of any AI solution.
Security must be present from day one. An agent that processes confidential data or writes SQL queries needs least privilege, auditing, and protection against injection attacks. Q2BSTUDIO offers cybersecurity services, including pentesting and architecture reviews, so that AI skills do not become an attack vector. An insecure data flow puts both information and customer trust at risk.
Another key piece is business intelligence. The reporting phase should not be limited to formatting text; statistical results should be integrated into dashboards and BI/Power BI systems. That requires clean data models and automated processes that connect analysis with visualization. Q2BSTUDIO helps build that bridge, preventing the knowledge generated by AI from remaining trapped in isolated documents.
The practical recommendation for a data team is clear: before adopting an LLM-generated skill, run a controlled experiment with representative tasks. Compare the result with the base prompt, measure quality, time, and cost. Then decide whether that skill deserves to be refined. Most organizations do not need thousands of skills; they need a few well-designed and verified workflows.
Moreover, the combination of people and machines is still essential. A data scientist can turn an automatic skill into a reliable instrument by reviewing, correcting, and adapting it to their context. Companies that understand this gain competitive advantage: they use AI to accelerate, but they do not give up human judgment. Automation should not be understood as replacing experience, but as amplifying it.
In short, the recent empirical finding invites caution: LLM-generated skills do not improve results simply by existing. They are not a universal prompting strategy. Real progress will come when organizations integrate AI, cloud, cybersecurity, and BI into a solid software platform. Q2BSTUDIO, with its experience in custom applications and intelligent agents, is ready to support that process with practical, results-oriented technology.
