- 1153 Views
- 2 replies
- 0 kudos
Resolved! How to handle load of 300 tables to delta lake
My task is to sync 300 tables from on prem sql server to delta lake. I will load CDC from Raw. First step is to move CDC data to bronze with autoloader. Then using delta stream get changes from bronze, make simple datatype changes and merge this data...
- 1153 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @alesventus ,You can apply metadata/config driven approch. You can create control table (or json/yaml file) with all information that are required for processing like:- table name- target table- table primary keys- transformation to applyAnd then ...
- 0 kudos
- 1753 Views
- 2 replies
- 3 kudos
Resolved! Process single data set with different JSON schema rows using Pyspark in databricks
Hi,i am getting data from event hub and stored in delta table as a row table, i data i received in json , the problem i data i have different schema in each row but i code i use it take first row a json schema i am stuck how to do please any one gui...
- 1753 Views
- 2 replies
- 3 kudos
- 3316 Views
- 9 replies
- 7 kudos
Resolved! Where exactly I should create Volume in a catalog?
Currently my Databricks looks like this: I want to create volume to access external location. Where exactly should I create it? Should a create new schema in "poe" catalog and create a volume inside it or create it in a existing schema? What is the b...
- 3316 Views
- 9 replies
- 7 kudos
- 1012 Views
- 2 replies
- 3 kudos
How to parse VARIANT type column using Pyspark sintax?
I trying to parse VARIANT data type column, what is the correct sintax to parse sub columns using Pyspark, is it possible?.I'd like to know how to do it this way (I know how to do it using SQL syntax).
- 1012 Views
- 2 replies
- 3 kudos
- 3 kudos
As an addition to what @szymon_dybczak already said correctly. It's actually not a workaround, it's designed and documented that way. Make sure that you understand the difference between `:`, and `.`.Regarding PySpark, the API has other variant relat...
- 3 kudos
- 3563 Views
- 6 replies
- 7 kudos
Make the job fail if a task fail
Hi everyone,I have a job with 2 tasks running independently. If one of them fails, the remaining task continues to run. I would like the job to fail if any task fails.Is there any way to do that?Thank you!
- 3563 Views
- 6 replies
- 7 kudos
- 7 kudos
Extending to what @mhiltner has suggested, let's say you have 2 streaming tasks streamA and streamB. Create 2 separate tasks taskA and taskB. Each of these tasks should execute the same notebook which makes an API call to the CANCEL RUN or CANCEL AL...
- 7 kudos
- 18452 Views
- 4 replies
- 3 kudos
PermissionError: [Errno 1] Operation not permitted: '/Volumes/mycatalog'
We are having intermittent errors where a Job Task cannot access a Catalog through a Volume, with the error: `PermissionError: [Errno 1] Operation not permitted: '/Volumes/mycatalog'`.The Job has 40 tasks running in parallel and every few runs we exp...
- 18452 Views
- 4 replies
- 3 kudos
- 3 kudos
It appears to be a concurrency limitation, and there were fixes in the past but there is a possibility it may be a new code flow, adding a retry to the operation can mitigate the issue and work as a workaround. But you can report the issue with Datab...
- 3 kudos
- 7284 Views
- 2 replies
- 0 kudos
Cluster termination issue
I am using Databricks as a Community Edition user with a limited cluster (just 1 Driver: 15.3 GB Memory, 2 Cores, 1 DBU). I am trying to run some custom algorithms for continuous calculations and writing results to the delta table every 15 minutes al...
- 7284 Views
- 2 replies
- 0 kudos
- 0 kudos
If you set the "Terminate after" setting to 0 minutes during the creation of an all-purpose compute, it means that the auto-termination feature will be turned off. This is because the "Terminate after" setting is used to specify an inactivity period ...
- 0 kudos
- 4550 Views
- 4 replies
- 4 kudos
Run multiple jobs with different source code at the same time with Databricks asset bundles
Hi,I am migrating from dbx to databricks asset bundles. Previously with dbx I could work on different features in separate branches and launch jobs without issue of one job overwritting the other. Now with databricks asset bundles it seems like I can...
- 4550 Views
- 4 replies
- 4 kudos
- 4 kudos
We have the same issue. We might have multiple open PR on the bundles that are deploying the code, pipelines, jobs, etc. to the same workspace before the merge and they keep overwriting each other in the workspace.The jobs already have a separate ID ...
- 4 kudos
- 1062 Views
- 3 replies
- 2 kudos
can we parameterize the tags in the job compute
I want to monitor the cost better for the databricks job computes.I am using tags in the cluster to monitor cost.The tag values is static as of now.can we parameterize the compute the job cluster so that I can pass the tag values during the runtime a...
- 1062 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @,If you're using ADF you can look at below article:Applying Dynamic Tags To Databricks Job Clusters in Azure Data Factory | by Kyle Hale | MediumIf not, I think you can try to write some code that will use below endpoint. The idea is, before exec...
- 2 kudos
- 555 Views
- 0 replies
- 0 kudos
Partition In Spark with subqeury which include Union
I have a SQL query like this:select ... from table1 where id in (slect id from table 1 where (some condition) UNION select id from table2 where (some condition)) table1I have made a partition of 200 where upper bound is 200 and lower bound is 0 and p...
- 555 Views
- 0 replies
- 0 kudos
- 1672 Views
- 3 replies
- 3 kudos
Resolved! Databricks workflow each task cost
Suppose if we have 4 tasks (3 notebooks and 1 normal python code) in a workflow then i would like to know the cost incurred for each task in the Databricks workflow. Please let me know the any way to find out this details.
- 1672 Views
- 3 replies
- 3 kudos
- 3 kudos
If each of the tasks are sharing the same cluster then no, you cannot differentiate the costs between the tasks. However, if you setup each task to have its own job cluster, then pass some custom tags and you can then differentiate/report the costs ...
- 3 kudos
- 593 Views
- 0 replies
- 0 kudos
Confuse about large memory usage of cluster
We set up a demo DLT pipeline with no data involved: @Dlt.table( name="demo" ) def sample(): df = spark.sql("SELECT 'silver' as Layer") return df However, when we check the metric of the cluster, it looks like 10GB memory has already be...
- 593 Views
- 0 replies
- 0 kudos
- 682 Views
- 0 replies
- 0 kudos
Ingesting data from SQL Server foreign tables
I have created a connection to a SQL server DB, and set up a catalog for it. i can now view all the tables, and query them. I want to ingest some of the tables into our ADLS gen 2 that we set up with Unity Catalog. What is the best approach here? Lak...
- 682 Views
- 0 replies
- 0 kudos
- 1043 Views
- 1 replies
- 0 kudos
Running jar on Databricks cluster from Airflow
Hello,I have a jar file which is installed on a cluster. I need to run this jar from Airflow using DatabricksSubmitRunOperator. I followed the standard instructions as available on Airflow docshttps://airflow.apache.org/docs/apache-airflow-providers-...
- 1043 Views
- 1 replies
- 0 kudos
- 1546 Views
- 0 replies
- 0 kudos
dbt writting into different schema
I have a unity catalog and it goes like `catalogname.schemaname1`& `catalogname.schemaname2`. and I am trying to write tables into schemaname2 with dbt, the current setup in the dbt profiles.yml is prj_dbt_databricks: outputs: dev: cata...
- 1546 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
