- 3602 Views
- 1 replies
- 0 kudos
Migrating jsonb data from Postgresql database to Databricks
Hi Team,I am trying to create a pipeline to incrementally ingest data from an RDS postgresql database which contains tables that have some columns of jsonb data type. I am currently using AWS DMS with CDC to first load the data into an S3 bucket as c...
- 3602 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Karene, You can do the transformation in following manner from string to struct and refer to the example below: data =[('001','{"name":"bhupendra","zipcode":"260100"}')]schema = ['id','propertytype']df = spark.createDataFrame(data,schema)df.sho...
- 0 kudos
- 7639 Views
- 4 replies
- 1 kudos
urgent: apply changes into delta live tables
Hi @Retired_mod , @Avnish_Jain How can I implement cdc in sql dlt pipelines with a live table (not streaming) . I am trying to implement below where i am reading from external tables, loading data into bronze layer and then want to apply these change...
- 7639 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Faisal, APPLY CHANGES INTO does not support a materialized view as a source, this must be a streaming table. Ideally, your bronze tables are append-only with the source providing data incrementally. If you do get revisions on previous records in y...
- 1 kudos
- 21883 Views
- 3 replies
- 4 kudos
Resolved! AttributeError: 'DataFrame' object has no attribute 'rename'
Hello, I am doing the Data Science and Machine Learning course. The Boston housing has unintuitive column names. I want to rename them, e.g. so 'zn' becomes 'Zoning'. When I run this command: df_bostonLegible = df_boston.rename({'zn':'Zoning'}, axi...
- 21883 Views
- 3 replies
- 4 kudos
- 4 kudos
If df_boston is a DataFrame, but you still face issues, try an alternative syntax: df_boston = df_boston.rename(columns={'zn': 'Zoning'}).Make sure df_boston is a proper DataFrame and you're using a recent version of Pandas.
- 4 kudos
- 3627 Views
- 1 replies
- 0 kudos
import * from ../my/relative/path
I have the following repository structure:/Repos/main/MyRepo/ -> run (folder) -> setup (folder) -> src (folder) -> main (notebook)Now, starting from a notebook in the "src" folder I need to run/import all variables defined in all notebooks stored in ...
- 3627 Views
- 1 replies
- 0 kudos
- 19598 Views
- 8 replies
- 0 kudos
error after updating delta table com.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update
Hi,I have pipeline running. I have updated one file in delta table which is already processed. Now I am getting errorcom.databricks.sql.transaction.tahoe.DeltaUnsupportedOperationException: Detected a data update. This is currently not supported. If ...
- 19598 Views
- 8 replies
- 0 kudos
- 1992 Views
- 0 replies
- 0 kudos
- 1992 Views
- 0 replies
- 0 kudos
- 3396 Views
- 2 replies
- 0 kudos
Materialize View creation not supported from DLT Pipeline
Hi Team, I have a very basic scenario where I am using my custom catalog and want materialize view to get created from DLT table at the end of pipeline. The SQL used as below for the same.where "loom_data_transform" is a Streaming table. But pipeline...
- 3396 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @abhijitnag are you sure your loom_data_transform was created as a STREAMING table? docs:https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html
- 0 kudos
- 2172 Views
- 0 replies
- 0 kudos
DLT || Python || Aggregate Functions recomputing all the records
Hi all, I am building a realtime dashboard using Databricks Delta Live Tables Pipeline and using the following steps : - Bronze Table : Using the autoloader functionality provided by databricks, its incrementally ingesting new files records into a br...
- 2172 Views
- 0 replies
- 0 kudos
- 1240 Views
- 0 replies
- 0 kudos
Registered for Wrong certification
Hi, I am registered for databricks certification today at 23:00hr and I accidentally choose databricks data analyst certification instead of databricks data engineer associate.I have left several mails and submitted several forms to be able to fix th...
- 1240 Views
- 0 replies
- 0 kudos
- 4400 Views
- 1 replies
- 1 kudos
Issue with Reading MongoDB Data in Unity Catalog Cluster
I am encountering an issue while trying to read data from MongoDB in a Unity Catalog Cluster using PySpark. I have shared my code below: from pyspark.sql import SparkSession database = "cloud" collection = "data" Scope = "XXXXXXXX" Key = "XXXXXX-YYY...
- 4400 Views
- 1 replies
- 1 kudos
- 1 kudos
Few points 1. chce if you installed exactly same driver version as you are pointing this in code (2.12:3.2.0) it has to match 100percentorg.mongodb.spark:mongo-spark-connector_2.12:3.2.02. I have seen people configuring connction to atlas in two way...
- 1 kudos
- 3152 Views
- 1 replies
- 0 kudos
Unable to locate "Configure databricks" in visual studio code
Need help in locating "configure databricks" option to connect to a cluster in visual studio code. Not sure which settings need to enabled to view/access the configuration. Below is the screen shot of it.
- 3152 Views
- 1 replies
- 0 kudos
- 0 kudos
- 0 kudos
- 2328 Views
- 0 replies
- 0 kudos
Dashboard HTML Export: DisplayHTML Formatting Lost
I am building an analytics dashboard that we plan to export into HTML to provide to clients. We are running into major issues specifically with DisplayHTML visualizations. We are using DisplayHTML for help text, sub-headings, and for a KPI banner at ...
- 2328 Views
- 0 replies
- 0 kudos
- 5050 Views
- 1 replies
- 0 kudos
DeltaRuntimeException: Keeping the source of the MERGE statement materialized has failed repeatedly.
I'm using Autoloader (in Azure Databricks) to read parquet files and write their data into the Delta table.schemaEvolutionMode is set to 'rescue'.In foreach_batch I do1) Transform of read dataframe;2) Create temp view based on read dataframe and merg...
- 5050 Views
- 1 replies
- 0 kudos
- 0 kudos
Hmm, you can't have duplicated data in source dataframe/batch but it should error out with diffrent erro like "Cannot perform Merge as multiple source rows matched and attempted to modify the same target row...".Also this behaviour after rerun is str...
- 0 kudos
- 4353 Views
- 1 replies
- 0 kudos
Slow stream static join in Spark Structured Streaming
SituationRecords are streamed from an input Delta table via a Spark Structured Streaming job. The streaming job performs the following.Read from input Delta table (readStream)Static join on small JSONStatic join on big Delta tableWrite to three Delta...
- 4353 Views
- 1 replies
- 0 kudos
- 0 kudos
You have quite small machines that you are using, please take into consideration that a lot of memory of machine is occupied by other processes https://kb.databricks.com/clusters/spark-shows-less-memoryThis is not good idea to broadcast huge data fra...
- 0 kudos
- 17356 Views
- 4 replies
- 3 kudos
Resolved! How to run code-formating on the notebooks
Has anyone found a nice way to run code-formating (like black) on the notebooks **in the workspace**? My current workflow is to commit the file, pull it locally, format, repush and pull. It would be nice if it was some relatively easy way to run blac...
- 17356 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi Erik,I don't know if you are aware of this feature, currently there is an option to format the code in your databricks notebooks using the black code style formatter.Just you need to either have a version of your DBR equal to or greater than 11.2 ...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
