Using ChatGPT with your own data: the complete guide for companies in Austria

There are three ways to use ChatGPT with your own data: by copying and pasting individual texts into the chat, via uploaded files and so-called custom GPTs, or via a RAG-based knowledge base that makes a company’s entire document stock searchable. For permanent use in a business, usually only the third route is practicable, because it alone combines large volumes of documents, ongoing updates and traceable sources. The approach delivers the greatest benefit where the real company knowledge lies: in instructions, service records, solutions to problems and the experience of long-serving employees. Which option fits depends on the volume of documents, the data protection requirements and the intended use.

In brief

• ChatGPT does not know your company filing system and cannot access a single internal document on its own.
• Copy and paste and file uploads work for individual cases, but fail on volume, currency and missing sources.
• RAG-based knowledge bases search the entire document stock and back up every answer with its source.
• The biggest lever is documented company know-how: instructions, service records, solutions to problems and project knowledge – not the management of records.
• The ERP knows what was sold – the knowledge base knows how it is done.
• For company data, an EU server location, a data processing agreement (DPA) and the exclusion of use for training are crucial.
• Getting started works best with a clearly defined document stock and a few typical questions to check the answers against.

Why doesn’t ChatGPT know your company’s filing system?

ChatGPT was trained on publicly accessible texts from the internet and has no access to your server, your cloud storage or your email mailbox. The underlying language model – an LLM (large language model, a computer program that has learned from huge amounts of text to understand and produce language) – therefore knows neither your installation instructions nor your service reports or project documents. If you ask ChatGPT how a particular system is set up in your own company or which special agreement applies with a customer, it can only guess or pass.

There is also a technical limit: the context window. This means the amount of text a language model can process at the same time in a single conversation. Even generous context windows hold only a fraction of what accumulates in documents over years in an established company. The glossary explains these and other technical terms relating to AI and document search in detail.

Another point concerns currency. A language model’s knowledge ends at the point of its training, while your company data changes every day. Without a connection to your real documents, the AI inevitably works with outdated or invented knowledge. That is exactly why ChatGPT seems so convincing on general questions and so helpless on company-specific ones: one source of knowledge is huge, the other is simply not connected.

What ways are there to use your own data with ChatGPT and AI?

Three approaches have emerged in practice, and they differ considerably in effort, capacity and reliability. The following overview shows the most important differences before each route is described in detail.

Criterion	Copy and paste into the chat	Custom GPT with files	RAG knowledge base
Volume of documents	Individual texts	Limited number of files	Entire stock
Currency	Only what is being pasted in	Maintained manually	Can be expanded continuously
Source	None	Unreliable	Source for every answer
Effort per question	High	Low	Low
Setup effort	None	Low	Higher, one-off
Suitability for the company	Individual cases	Clearly defined topics	Permanent operation

Copy and paste: quick, but laborious

With the simplest route, you copy the relevant text directly into the chat window and ask your question about it. For a single contract clause or a draft email, this works well and requires no setup at all.

Copy and paste is no good as a permanent solution. Before every question you have to know which document contains the answer – which leaves the real problem, searching, unsolved. In addition, every copy operation puts company data into a third-party system in an uncontrolled way, without any rules on what happens to it.

Custom GPTs and file uploads: good for clearly defined topics

ChatGPT lets you upload files and create so-called custom GPTs – preconfigured chats that are given a limited number of documents permanently. For a clearly defined topic, such as a product manual or a collection of templates, such an assistant is quite usable.

The limits show up quickly in everyday use. The number of files is restricted, new or changed documents have to be added by hand, and the answers often give no source or only an imprecise one. If you want to check whether a statement really appears in the original document, you have to search for it yourself.

RAG knowledge base: the route for permanent operation

RAG stands for retrieval-augmented generation – roughly: the AI first picks out the relevant passages in your documents and only then formulates the answer on that basis. The language model therefore does not answer from its training knowledge, but from your actual records. How such a system is structured in detail is described in the foundation article What is an AI knowledge base?.

The decisive advantage lies in the combination of volume and traceability. A RAG solution indexes the entire document stock – including scanned paper documents, provided text recognition (OCR, optical character recognition: the automatic conversion of images and scans into searchable text) is built in. Every answer refers to the source document, which can be opened and checked with one click.

To be honest, this also means: a RAG solution has to be set up and filled, and the quality of the answers stands or falls with the quality of the stored documents. If you have neglected your filing for years, AI alone will not give you order for free.

Which knowledge belongs in an AI knowledge base – and which does not?

The value of an AI knowledge base does not lie in managing invoices or delivery notes – that is what accounting and ERP (enterprise resource planning, the central software for orders, stock and billing) are for. Its value lies in the company’s documented know-how: in instructions, service records, documented solutions to problems, project and contract knowledge and in all the experience that until now has existed only in the heads of long-serving employees. In short: the ERP knows what was sold – the knowledge base knows how it is done.

This distinction is not hair-splitting; it determines the benefit. Transaction data is structured and well looked after in existing systems; nobody needs an AI to look up an invoice number. The real problem in day-to-day business is unstructured content: the Word document with the commissioning instructions from six years ago, the email thread in which a special solution was agreed with the customer, the handwritten service record that only the colleague who wrote it can find. It is exactly this knowledge that a RAG knowledge base makes queryable in plain language.

Type of knowledge	Typical examples	Belongs primarily in
Transaction data	Invoices, delivery notes, stock levels, order status	ERP or accounting
Structured master data	Customer addresses, item numbers, price lists	ERP or CRM
Documented know-how	Instructions, service records, checklists, solutions to problems	AI knowledge base
Project and contract knowledge	Specifications, meeting notes, special agreements	AI knowledge base
Experience in people’s heads	Tricks of long-serving employees, fault histories, customer peculiarities	Document first, then knowledge base

Records can certainly be included in a knowledge base – for example when an old quote provides the context for a special agreement. But they are an extra, not the core. How an AI knowledge base differs from a classic document management system, which specialises in the orderly, audit-proof filing of records, is shown in the comparison of document management and AI knowledge base.

How do you preserve the experience of long-serving employees?

A company’s most valuable knowledge is not in any folder: why the system at customer X shows the same fault every two years, which supplier really delivers when things are tight, how the calculation for custom work is structured. If an experienced member of staff retires or resigns, this knowledge often disappears without replacement – and their successors need years to build it up again. An AI knowledge base can preserve this knowledge, but only if it has been recorded in some form beforehand.

The good news: this does not require a documentation project with the ambitions of a manual. For AI-supported search, short, honest notes are enough, because the system finds the relevant passage even in keywords and informal texts. The following sequence has proved its worth:

1. Name the knowledge holders: Identify the two or three people who are asked most often in everyday work – these are your critical knowledge holders.
2. Document as the occasion arises: Agree that a short note is written after every special case that has been solved – what was the problem, what was the solution, what needs to be considered next time. Three sentences are enough.
3. Allow low-threshold formats: A voice memo, a photo of the rating plate with a comment or a keyword record are better than nothing at all. Text recognition makes photos and scans searchable too.
4. Structure handover conversations: If a retirement is coming up, hold several short conversations along specific guiding questions – important customers, tricky systems, recurring faults – and record the answers in writing.
5. Feed them into the knowledge base: File the notes centrally so that they are indexed, and check with real questions whether the content is found.
6. Establish a routine: Make recording solutions a fixed part of completing a job – otherwise the practice will fizzle out again after a few weeks.

The difference from the classic wiki-style knowledge base is significant: there, maintenance regularly fails because content has to be neatly structured, categorised and tagged. A RAG-based solution does not make this demand – it finds the short note from March just as well as the formatted record from January.

What do you need to consider regarding data protection?

Before company data goes into any AI system, three questions need to be clarified: where are the servers, is the data used for training, and is there a data processing agreement? The GDPR (General Data Protection Regulation, the EU-wide data protection law) requires that personal data – that is, all information about identifiable persons such as customers or employees – is processed only on a clear legal basis.

The server location determines which jurisdiction your data is subject to. Providers with servers in the EU make GDPR-compliant use considerably easier, while US services require additional legal arrangements.

The training question is at least as important. With free and private ChatGPT accounts, entries can by default be used to develop the models further unless you actively object. For company data, only access where use for training is excluded by contract comes into question.

The GDPR provides for a data processing agreement (DPA: the written agreement that governs how a service provider may handle your data) when personal data is processed by third parties. Reputable providers supply such an agreement on their own initiative; information on the legal basis is provided by the Austrian Data Protection Authority at dsb.gv.at. You will find a detailed look at the legal situation in Austria, including a practical checklist, in the article on AI and the GDPR in Austria.

How can access within the company be regulated securely?

Data protection towards the outside world is not the end of it – internally, too, handling a searchable knowledge base needs clear rules. Without a permissions concept, an AI that knows the entire document stock would answer everything for everyone: including questions about salaries, cost calculations or personnel files. That must not happen.

The central tool is role and permission management. It defines which person may query which areas of documents, and it does so at the level of the search itself: the system may only use documents for the answer that the person asking would also be allowed to open directly. A fitter thus gets access to service records and instructions, but not to the contribution margin calculation; accounting sees contracts, but not job applications.

Three further points belong to a solid internal security concept. First, logging: it should be traceable who made which queries and when – not to monitor the workforce, but to be able to investigate misuse if the need arises. Second, offboarding: if someone leaves the company, their access must end on their last working day, as naturally as handing back the key. Third, avoiding shadow AI: if the company does not offer a regulated tool, experience shows that employees use private ChatGPT accounts – and then company data flows uncontrolled into systems to which none of the protective measures mentioned apply. An officially introduced, data-protection-compliant system is therefore also a protective measure against unofficial use. The Austrian Economic Chamber also offers guidance on organisational questions of digitalisation at wko.at.

What does an SME realistically need – and what not?

A small or medium-sized company as a rule does not need its own self-trained AI. Training or fine-tuning your own models is expensive, maintenance-intensive and simply the wrong tool for the purpose “answers from our own documents” – new documents would only be taken into account after retraining.

What an SME actually needs is less spectacular: a central place where the documented company knowledge comes together in searchable form, queries in plain language instead of keyword guessing, and answers backed up with a source. On top of that come practical requirements such as capturing scans and photos, a connection to existing filing systems and tools such as network drives or Microsoft 365, and user management so that not everyone sees everything.

Just as important is what is not needed: no data export into uncontrolled systems, no rebuild of the entire IT and no months of staff training. If using it feels like a familiar chat, the threshold is low even for employees who are not particularly technically minded. Nor does anything need replacing: ERP, accounting and industry software remain what they are – the knowledge base sits as a search layer over what these systems do not cover.

How do you implement it step by step?

Introducing a document-based AI is not a major IT project; it can be done in manageable stages. The following order has proved its worth:

1. Collect use cases: For one week, note what people in the company actually search for and who has to interrupt whom to find it – these questions will later be used to check the answers.
2. Review the document stock: Decide which folders, drives and mailboxes promise the greatest benefit. Start with know-how – instructions, records, project folders – not with the archive of records. Completeness is not the goal at the beginning.
3. Check the provider and data protection: Clarify the server location, exclusion of training and DPA before the first document is uploaded.
4. Initial filling and checking the answers: Fill the system with the starting stock and check the answers against the real questions you collected – including checking the sources.
5. Training and routine: Show the team how to use it with specific examples and build the filing of new documents and solution notes into the daily routine.

If you would rather not go down this route alone, you can find structured support: Goma-it’s AI knowledge base was developed for exactly this purpose in Austrian SMEs, with EU servers, a source for every answer and a guided introduction process from the initial conversation to training.

Which tasks does AI with your own data solve in everyday work?

The benefit shows most clearly in the small, constantly recurring knowledge questions. The fitter on the building site asks from their smartphone how the fault code on a particular type of system was fixed last time – and is shown the relevant passage from the service record at the time, without having to call the office. The settings from the commissioning documentation from years ago can be retrieved just as easily as a colleague’s note that the caretaker must be informed first at this customer.

In the back office, the questions are often about projects and contracts: what was agreed with the customer on warranty, which special solution was chosen on the last comparable project, which arguments tipped the balance back then? Such answers are hidden in meeting notes, email threads and specifications – documents that are practically impossible to find without a searchable knowledge base as soon as the responsible colleague is not available.

The induction of new employees also changes noticeably. Anyone who can put questions about a customer’s history or internal procedures directly to the knowledge base has to interrupt colleagues less often – and experienced staff gain time for the cases that really need experience. Incidentally, these constant interruptions are a considerable, usually invisible cost factor; the article Time lost searching for documents looks at this in detail.

Which mistakes and misunderstandings are most common at the start?

Introduction projects reveal some typical stumbling blocks that can be avoided with a little foresight.

The most common mistake is the ambition to capture everything at once. Anyone who uploads the complete archive including twenty years of records at the start dilutes the quality of the hits and delays the first visible benefit. A curated starting stock made up of the knowledge that people actually ask about makes more sense.

The second most common misunderstanding is mistaking it for a replacement for an ERP or DMS. A knowledge base does not post invoices or keep stock levels – and it is not supposed to. Anyone who measures it against this expectation will be disappointed; anyone who understands it as a search layer over unstructured company knowledge gets exactly what the systems alongside it lack.

The inertia of documentation habits is also often underestimated. The system can only find what has been filed – if the solution note after a special case is not written, the knowledge base stands still too. Without a named person responsible for maintaining the stock and reminding the team, even the best introduction will fizzle out.

A fourth point concerns the choice of tool: company data in private ChatGPT accounts is not a trivial offence but a real data protection problem – without a DPA, without exclusion of training, without control. And finally, exaggerated expectations of the AI itself: it finds and provides evidence, but it does not tidy up a chaotic filing system and does not decide which of two contradictory document versions applies.

How can you tell whether getting started is worthwhile for your company?

Whether an AI knowledge base makes sense for a particular company can be judged from a few everyday observations. Clear signals are: the same two or three people are constantly interrupted with knowledge questions because only they know where something is or how something works. New employees take noticeably long to work independently. Experienced staff whose knowledge is documented nowhere are due to retire in the next few years. And: there are several possible places where one and the same piece of information might be filed – network drive, mailbox, paper folder, a chat history.

If two or more of these points apply, the pressure is usually high enough to make a structured check worthwhile. If none applies – for example because the company is very small, the knowledge lies with a single person who will be staying for a long time and the filing is kept lean – the decision can be postponed with a clear conscience. An honest assessment of where you stand is more valuable than buying a tool for which there is no real need yet.

For the check itself, the rule of thumb from the step-by-step guide above applies: collect real questions from everyday work, fill a clearly defined stock, check the answers including their sources. After a few weeks it becomes reliably clear whether the system holds up in your particular company.

What are the limits of AI with your own data?

Even the best document-based AI only answers what is in the documents. Knowledge that exists exclusively in employees’ heads remains invisible until it is recorded in some form – even if only as a short note. That is precisely why the knowledge preservation described above is not a sideshow but half the battle.

Contradictory or outdated documents remain a content problem. The AI can show which document a statement comes from, but cannot always judge which of two differing versions is the valid one. Basic order in the filing system, such as marking superseded versions, remains the company’s job.

Finally, the AI does not replace professional judgement. It finds the relevant contract clause quickly, but whether that leads to a termination, an additional claim or a call to the lawyer is still decided by a person. Used correctly, the technology takes the searching off the team’s hands – not the thinking.