Protecting personally identifiable information (PII) has become a strategic priority for companies across all industries, especially those operating in regulated environments such as finance, insurance, healthcare or government. The exponential growth of data volumes and the diversity of formats make manual classification methods unsustainable. In this context, Amazon Macie combined with AWS Step Functions offers an automated and scalable solution to detect PII in a customized way, adapting to the specific needs of each organization.
However, Macie's standard detection covers common patterns like social security numbers, email addresses or credit card numbers, but does not include business-specific identifiers such as internal customer codes, policy numbers or record identifiers. To fill this gap, it is necessary to implement custom data identifiers that allow the platform to recognize those exclusive formats. At Q2BSTUDIO, as a company specialized in custom software development, we know that personalization is key for cybersecurity and compliance solutions to truly align with business operations.
In this article we will explore how to build an automated pipeline that detects PII in real time every time a file arrives in Amazon S3, using Amazon EventBridge, AWS Step Functions, Amazon Macie with custom identifiers, and Amazon SNS for notifications. The architecture follows a three-bucket pattern to maintain traceability and data segregation, and generates audit-ready reports in CSV and JSON formats.
The flow starts when an object is uploaded to the 'raw' bucket. An EventBridge event triggers the execution of a Step Functions state machine. This orchestration copies the file to a 'staging' bucket (with automatic expiration after 7 days), creates a Macie classification job including both AWS managed identifiers and custom identifiers defined by the company, waits for the analysis to finish (with configurable retries and wait states), retrieves the findings, generates timestamped reports, and sends SNS notifications if critical data is detected. Finally, the original file is moved to the 'scanned' bucket and the temporary copy is deleted.
One of the most important advantages of this solution is the ability to incorporate custom identifiers through regular expressions (regex). For example, an insurance company can define a pattern for policy numbers like 'POL-\d{6,10}', or a hospital for patient identifiers like 'PAT-\w{8}'. These patterns are deployed as part of the CloudFormation stack, enabling centralized and versioned management. Additionally, up to 10,000 custom identifiers can be defined per AWS account, although each classification job can use a maximum of 30, so for high volumes it is advisable to distribute identifiers across multiple jobs.
Integration with additional AWS services further enhances the solution. Findings reports can be sent to Amazon Athena or AWS Glue for trend analysis, and then visualized in Business Intelligence dashboards with Power BI. It is also possible to connect findings to AWS Security Hub for unified security management. Even automated remediation flows can be implemented, such as moving files with high-severity PII to a quarantine zone or temporarily revoking access permissions.
For organizations that handle data across multiple AWS accounts, the solution can be extended using cross-account EventBridge rules, centralizing scanning in a dedicated security account. This is especially relevant when operating in multi-tenant environments where data segregation is critical. At Q2BSTUDIO, we also recommend enabling SSE-KMS encryption on buckets and SNS topics, as well as logging all access via CloudTrail to maintain a complete audit trail.
Another aspect to consider is performance and cost optimization. The presented design creates one Macie job per object, providing real-time detection but potentially generating high API call frequency. For massive workloads, the TriggerScan logic can be modified to batch multiple objects into a single job, respecting the 0.1 requests per second limit. Likewise, it is advisable to monitor Macie quotas and adjust the number of custom identifiers per job as needed.
Implementing this pipeline brings numerous benefits to compliance and security teams. Compliance teams obtain detailed reports with timestamps that demonstrate where PII resides and how it has been handled, facilitating audits and demonstrating compliance with regulations such as GDPR, HIPAA, CCPA or PCI DSS. Security teams receive immediate alerts on critical findings, enabling rapid response to potential data breaches.
At Q2BSTUDIO, as a software and technology development company, we offer consulting and implementation services for cloud solutions on AWS and Azure, integrating capabilities of artificial intelligence and process automation to maximize data value. Automated PII detection is just one example of how combining managed services with custom logic can solve complex problems efficiently and securely.
To get started, it is recommended to deploy the CloudFormation stack in a test environment, customize the identifiers with business-specific patterns, and test with sample files. Once validated, hardening measures for production can be applied: encryption, access control, logging and scaling. The architecture's flexibility allows adaptation to different volumes and regulatory requirements.
In conclusion, automating PII detection with Amazon Macie and Step Functions not only improves the organization's security posture, but also frees human resources from repetitive and error-prone tasks. By including custom identifiers, the solution becomes truly business-specific, closing the gap left by generic detectors. If your organization is looking to implement a robust and scalable data protection strategy, having a technology partner like Q2BSTUDIO can make a difference in adoption speed and quality of results.
