Hi there, I hope I have what is effectively a simple question. I'd like to ask for a bit on guidance if I am structuring my source-to-bronze auto loader data properly. Here's what I have currently:/adls_storage/<data_source_name>/<category>/autoloade...
As part of my MLOps stack, I have developed a few packages which are the published to a private AWS CodeArtifact repo. How can I connect the AWS CodeArtifact repo to databricks? I want to be able to add these packages to the requirements.txt of a mod...
One way to do it is to run this line before installing the dependencies:pip config set site.index-url https://aws:$CODEARTIFACT_AUTH_TOKEN@my_domain-111122223333.d.codeartifact.region.amazonaws.com/pypi/my_repo/simple/But can we add this in MLFlow?
Can I run multi query in one command using databricks JDBC driver and would databricks execute one query faster then running multi queries in one script?
Hi @shkelzeen, You can run multiple queries in one command using Databricks JDBC driver by using the operator to chain multiple SQL statements together. This will execute two queries in one command and display the results in separate tables. ...
Dear Databricks Community Members:The symptom: The DLT pipeline was failed with the error message: Failure to initialize configuration for storage account storageaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account...
Hi @harvey-c, One common issue is using the container names instead of the storage accounts in the URL, or including the container names in the configs. Ensure you use the correct storage account name and key in your code and settings.Another comm...
Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...
Quick follow-up on this @Kaniz_Fatma (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...
Hi all, we have a issue while trying to write a quite large data frame, close to 35 million records. We try to write it as parquet and also table and none work. But writing a small chink (10k records) is working. Basically we have some text on which ...
That could work, but you will have to create a UDF.Check this SO topic for more info
Hello - I registered for the Databricks Data Engineering Associate Certification exam. I hit an issue, their Secure browser would not launch, it just crashed - the only thing I could see in a flash is "bad request" and poof its gone. Spend over 2 h...
Hi @vrajesh123, Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 ho...
Hi All,I have a requirement in my project, where we will be writing some python code inside databricks . Please note we will not be using pyspark . It will plain pythin with polars.I am looking into ho to create test files for main file. Below is sim...
Hi @sudhanshu1 , The error you're encountering is due to the fact that pytest is looking for a fixture named 'name' which is not defined. Fixtures in pytest are functions that are run before each test function to which it is applied. Fixtures are us...
Hi all,I was able to deploy a wheel to the /Shared/ folder from a repository in Gitlab with asset bundles. The databricks.yml looks something like this.artifacts: default: type: whl build: poetry build path: . targets: workspace: h...
Finally I decided to use AWS Code Artifact and mirror the PyPI, which I think it's a bit cleaner. But your solution looks good too. Thanks!
Hi TeamHow can we call informatica jobs from data bricks? could you please suggest on this.Regards,Phanindra
Hi @Phani1, Informatica and Databricks have a partnership that allows for seamless integration between the two platforms. Here’s a high-level overview of how you can use Informatica with Databricks: Data Ingestion: You can ingest data directly in...
How can I turn off displaying all this config text after running cells? I don't need to see all this text in every cell unless there is an error and it causes readability issues and makes the notebook scrolling tedious.
Needed some help with Time Travel with Delta Live tables We were trying to figure out if we can go in and alter the history on this table, and what would happen to data that we mass upload? By this we mean we have data from the past that we would ...
Delta Live Tables leverage Delta Lake, or Delta Tables. Delta tables, through transactions (e.g. insert, update, delete, merges, optimization) create versions of said Delta Table. Once a version is created it cannot be altered, it is immutable. Yo...
Hi Team,We would like to upgrade from Spark version 3.2 to 3.4+ (Databricks Runtime - 10.4 to 12.2/13.3)We would like to understand how complex upgradation is this and challenges which we face? what are the technical steps and precautions we need to ...
Hi @Phani1, Check Databricks Runtime release notes for any specific changes, improvements, or deprecations that might impact your workloads. Our End-of-Year Community Survey is here! Please take a few moments to complete the survey. Your feedback m...
Hi team,I'm struck in a Spark Structured streaming use-case.Requirement: To read two streaming data frames, perform a left join on it and display the results. Issue: While performing a left join, the resultant data frame contains only rows where ther...
Hi @RamanP9404, In Spark Structured Streaming, watermarking is essential for handling late data and ensuring correctness in stream-stream joins. When performing a left outer join, both the left and right datasets must be watermarked with a time const...
Hi Databricks Team,Could you please provide me the detailed steps on how to be enabled customer managed keys in databricks (AWS) Account, if there is any video on it that would be great helpful.Regards,Phanindra
Hi @Phani1, Enabling customer-managed keys (CMKs) in Databricks on AWS involves securing and controlling access to encrypted data. Here are the steps to set up CMKs: Prerequisites: Ensure your Databricks workspace is on the E2 version of the Dat...
