- 11986 Views
- 5 replies
- 0 kudos
Optimize & Compaction
Hi,From which data bricks runtime will support Optimize and compaction
- 11986 Views
- 5 replies
- 0 kudos
- 0 kudos
Optimize and compaction are operations commonly used in Apache Spark for optimizing and improving the performance of data storage and processing. Databricks, which is a cloud-based platform for Apache Spark, provides support for these operations on v...
- 0 kudos
- 2574 Views
- 1 replies
- 0 kudos
Resolved! Does repos support Ipynb ?
- 2574 Views
- 1 replies
- 0 kudos
- 0 kudos
Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can ...
- 0 kudos
- 2179 Views
- 1 replies
- 0 kudos
- 2179 Views
- 1 replies
- 0 kudos
- 0 kudos
I recorded a video showing how to setup SCIM in the account console. https://www.youtube.com/watch?v=AWPGQJRK6lo&t=1s
- 0 kudos
- 3196 Views
- 1 replies
- 0 kudos
- 3196 Views
- 1 replies
- 0 kudos
- 0 kudos
Feel free to read my medium blog where I summarized all the featureshttps://medium.com/@youssefmrini/databricks-workflows-features-oriented-3f9ec025301a
- 0 kudos
- 2857 Views
- 3 replies
- 0 kudos
Didn't receive my Databricks Spark 3.0 certification
I took the exam yesterday and passed the test. I haven't received any email from Databricks Academy. How long would it take to receive the certification?
- 2857 Views
- 3 replies
- 0 kudos
- 4205 Views
- 2 replies
- 0 kudos
Incorrect error when adding an IP access list
I have disabled the IP Access List on my workspace and am trying to add an IP list through the IP Access List API. However, when adding a list, I get the INVALID_STATE response.The docs mention this is because:"If the new list would block the calling...
- 4205 Views
- 2 replies
- 0 kudos
- 0 kudos
"One possible workaround could be to (1) temporarily enable the IP Access List feature, (2) add the necessary IP addresses to the list, and then (3) disable the feature again. This way, you can add the IP addresses you need without blocking the curre...
- 0 kudos
- 3388 Views
- 1 replies
- 0 kudos
Resolved! Delta tables background
Hi,Looking at the delta tables and how they are stored I have a question. If the delta tables are stored as parquet files in ADLS why if I copy/paste/rename a folder that corresponds to an existing table in the same location/database this does not ge...
- 3388 Views
- 1 replies
- 0 kudos
- 0 kudos
@Sara Corral​ :When you copy/paste/rename a folder that corresponds to an existing delta table in the same location/database, it does not generate a copy of the previous table because delta tables are not just plain parquet files. They have additiona...
- 0 kudos
- 9557 Views
- 1 replies
- 0 kudos
Databricks Cluster Autoscaling
Hello All,Could anyone please suggest impact of Autoscaling in cluster cost ?Suppose if I have a cluster where min worker is 2 and max is 10 but most of the time active worker are 3 so the cluster will be billed for only 3 workers or for 10 worker(...
- 9557 Views
- 1 replies
- 0 kudos
- 0 kudos
@Deepak Bhatt​ :Autoscaling in Databricks can have a significant impact on cluster cost, as it allows the cluster to dynamically add or remove workers based on the workload.In the scenario you described, if the active worker count is consistently at ...
- 0 kudos
- 22928 Views
- 1 replies
- 0 kudos
Populating data from databricks to sql server tables
Hello All,Could any one please suggest what is the best way to populate(Upsert) data from delta table into the sql server table.we are transforming our data in Databricks and storing data into the delta table. but for reporting purpose we need to pop...
- 22928 Views
- 1 replies
- 0 kudos
- 0 kudos
@Deepak Bhatt​ :Yes, using the Spark Synapse connector could be a good option for upserting data from a Delta table into a SQL Server table. The Spark Synapse connector allows you to read and write data from Azure Synapse Analytics, formerly known as...
- 0 kudos
- 8990 Views
- 11 replies
- 8 kudos
Resolved! Facing issues in running the converted code in spark sql framework with 5 to 10 percent volume of prod data. Need help in solving this and suggestions required.
Hi All.. Need your help in this issue what i am facing. Currently we are using data bricks as a platform to build pipeline and execute our talend ETL sqls converted into the spark sql framework as we were facing issues in loading the history data int...
- 8990 Views
- 11 replies
- 8 kudos
- 5431 Views
- 3 replies
- 3 kudos
SSO Implementation - User need to go through multiple steps to visit the "Select Workspace" page.
Below are the steps we have implemented to login through SSO.1. We have setup SSO and are able to login into Databricks using IDP (SiemensID Authentication). 2. After successful authentication, we have done the configuration of redirecting user to da...
- 5431 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @Kaniz Fatma​ @Debayan Mukherjee​ Thanks for the response.We have raised this issue with Databricks team internally. We have shared the details with team. I will post the solution once we find any breakthrough to resolve it.
- 3 kudos
- 3993 Views
- 3 replies
- 4 kudos
Issue while reading data from Kafka topic to Spark strutured streaming
py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.SQLContext.readStream() is not whitelisted on class class org.apache.spark.sql.SQLContextI already disable acl for cluster using "...
- 3993 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Ravi Teja​,Just a friendly follow-up. Do you still need help? if you do, please share more details, like DBR version, standard or High concurrency cluster? etc
- 4 kudos
- 4417 Views
- 2 replies
- 3 kudos
Academy Accreditation - SQL Analyst Associate
Hi Team , So last year I acquired SQL Analyst Associate badge and due for renew this Jan 2023 . However when checked in Databricks Academy couldn't find the course . So has it been retired or removed ? If exists can someone help me with the course d...
- 4417 Views
- 2 replies
- 3 kudos
- 2384 Views
- 3 replies
- 0 kudos
Custom duty charges or any other additional charges on Databricks rewards.
Hi All,Anyone can please confirm i have to pay any custom duty fee or ​any other additional shippment fee for Databricks rewards?​Thanks​
- 2384 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @S S​,Just a friendly follow-up. Do you still need help with this question? please let us know
- 0 kudos
- 11272 Views
- 6 replies
- 0 kudos
Resolved! Databricks Delta Live Table stored as SCD 2 is creating new records when no data changes. How do I stop this?
I have a streaming pipeline that ingests json files from a data lake using autoloader. These files are dumped there periodically. Mostly the files contain duplicate data, but there are occasional changes. I am trying to process these files into a dat...
- 11272 Views
- 6 replies
- 0 kudos
- 0 kudos
For clarity, here is the final code that avoids duplicates, using @Suteja Kanuri​ 's suggestion:import dlt @dlt.table def currStudents_dedup(): df = spark.readStream.format("delta").table("live.currStudents_ingest") return ( df.drop...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
