- 1575 Views
- 2 replies
- 0 kudos
Auto Loader Use Case Question - Centralized Dropzone to Bronze?
Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...
- 1575 Views
- 2 replies
- 0 kudos
- 0 kudos
Quick follow-up on this @Retired_mod (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...
- 0 kudos
- 1822 Views
- 3 replies
- 0 kudos
Fail to write large dataframe
Hi all, we have a issue while trying to write a quite large data frame, close to 35 million records. We try to write it as parquet and also table and none work. But writing a small chink (10k records) is working. Basically we have some text on which ...
- 1822 Views
- 3 replies
- 0 kudos
- 0 kudos
That could work, but you will have to create a UDF.Check this SO topic for more info
- 0 kudos
- 4102 Views
- 1 replies
- 0 kudos
Webassessor Secure Browser will not Launch during exam.
Hello - I registered for the Databricks Data Engineering Associate Certification exam. I hit an issue, their Secure browser would not launch, it just crashed - the only thing I could see in a flash is "bad request" and poof its gone. Spend over 2 h...
- 4102 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello - I registered for the Databricks Data Engineering Associate Certification exam. I hit an issue, their Secure browser would not launch, it just crashed - the only thing I could see in a flash is "bad request" and poof its gone. Spend over 2 h...
- 0 kudos
- 8252 Views
- 2 replies
- 2 kudos
Python file testing using pytest
Hi All,I have a requirement in my project, where we will be writing some python code inside databricks . Please note we will not be using pyspark . It will plain pythin with polars.I am looking into ho to create test files for main file. Below is sim...
- 8252 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi All,I have a requirement in my project, where we will be writing some python code inside databricks . Please note we will not be using pyspark . It will plain pythin with polars.I am looking into ho to create test files for main file. Below is sim...
- 2 kudos
- 2456 Views
- 1 replies
- 0 kudos
Asset bundle build and deploy python wheel with versions
Hi all,I was able to deploy a wheel to the /Shared/ folder from a repository in Gitlab with asset bundles. The databricks.yml looks something like this.artifacts: default: type: whl build: poetry build path: . targets: workspace: h...
- 2456 Views
- 1 replies
- 0 kudos
- 0 kudos
Finally I decided to use AWS Code Artifact and mirror the PyPI, which I think it's a bit cleaner. But your solution looks good too. Thanks!
- 0 kudos
- 859 Views
- 0 replies
- 0 kudos
informatica jobs from data bricks
Hi TeamHow can we call informatica jobs from data bricks? could you please suggest on this.Regards,Phanindra
- 859 Views
- 0 replies
- 0 kudos
- 497 Views
- 0 replies
- 0 kudos
How to suppress text output and just show the results in each cell?
How can I turn off displaying all this config text after running cells? I don't need to see all this text in every cell unless there is an error and it causes readability issues and makes the notebook scrolling tedious.
- 497 Views
- 0 replies
- 0 kudos
- 1299 Views
- 2 replies
- 0 kudos
time travel with DLT
Needed some help with Time Travel with Delta Live tables We were trying to figure out if we can go in and alter the history on this table, and what would happen to data that we mass upload? By this we mean we have data from the past that we would ...
- 1299 Views
- 2 replies
- 0 kudos
- 0 kudos
Delta Live Tables leverage Delta Lake, or Delta Tables. Delta tables, through transactions (e.g. insert, update, delete, merges, optimization) create versions of said Delta Table. Once a version is created it cannot be altered, it is immutable. Yo...
- 0 kudos
- 1363 Views
- 1 replies
- 0 kudos
Upgrade Spark version 3.2 to 3.4+
Hi Team,We would like to upgrade from Spark version 3.2 to 3.4+ (Databricks Runtime - 10.4 to 12.2/13.3)We would like to understand how complex upgradation is this and challenges which we face? what are the technical steps and precautions we need to ...
- 1363 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Team,We would like to upgrade from Spark version 3.2 to 3.4+ (Databricks Runtime - 10.4 to 12.2/13.3)We would like to understand how complex upgradation is this and challenges which we face? what are the technical steps and precautions we need to ...
- 0 kudos
- 1699 Views
- 0 replies
- 0 kudos
Customer Managed Keys in Databricks (AWS)
Hi Databricks Team,Could you please provide me the detailed steps on how to be enabled customer managed keys in databricks (AWS) Account, if there is any video on it that would be great helpful.Regards,Phanindra
- 1699 Views
- 0 replies
- 0 kudos
- 1923 Views
- 0 replies
- 0 kudos
Spark Streaming Issues while performing left join
Hi team,I'm struck in a Spark Structured streaming use-case.Requirement: To read two streaming data frames, perform a left join on it and display the results. Issue: While performing a left join, the resultant data frame contains only rows where ther...
- 1923 Views
- 0 replies
- 0 kudos
- 1132 Views
- 1 replies
- 0 kudos
Load GCP data to Databricks using R
I'm working with Databricks and Google Cloud in the same project. I want to load specific datasets stored in GCP into a R notebook in Databricks. I currently can see the datasets in BigQuery. The problem is that using the sparklyr package, I'm not ab...
- 1132 Views
- 1 replies
- 0 kudos
- 0 kudos
I'm working with Databricks and Google Cloud in the same project. I want to load specific datasets stored in GCP into a R notebook in Databricks. I currently can see the datasets in BigQuery. The problem is that using the sparklyr package, I'm not ab...
- 0 kudos
- 801 Views
- 0 replies
- 1 kudos
Best AI Art Generator
AI art generator uses artificial intelligence to create captivating artworks, redefining the boundaries of traditional creativity and enabling endless artistic possibilities.AI photo restoration is a groundbreaking technology that employs artificial ...
- 801 Views
- 0 replies
- 1 kudos
- 5559 Views
- 0 replies
- 0 kudos
Alter table
Hi Team, Could you please suggest, Do we have an alternate approach to alter the table instead of creating a new table and copying the data as part of the deployment.
- 5559 Views
- 0 replies
- 0 kudos
- 1091 Views
- 1 replies
- 0 kudos
Constraint options usage
I am trying to use the constraints options:NOT ENFORCEDDEFERRABLEINITIALLY DEFERREDNORELYHowever it seems I am not able to use them successfully. When I try to use them with PRIMARY KEYS (not sure if it is possible), I am not able to enforce any key....
- 1091 Views
- 1 replies
- 0 kudos
- 0 kudos
BTW the forum is bugged I can't paste code..
- 0 kudos
