- 7791 Views
- 2 replies
- 2 kudos
Bronze silver gold layers
Is there a best practise guide on setting up the delta lake for these 3 layers. ​I'm looking for document or scripts to run that will assist me.
- 7791 Views
- 2 replies
- 2 kudos
- 2 kudos
hi @Lloyd Vickery​ ,I would highly recommend to use Databricks Delta Live Tables (DLT) docs here https://databricks.com/product/delta-live-tables
- 2 kudos
- 6685 Views
- 3 replies
- 0 kudos
Try catch multiple write streams on a job
We are having issues with checkpoints and schema versions getting out of date (no idea why), but it causes jobs to fail. We have jobs that are running 15-30 streaming queries, so if one fails, that creates an issue. I would like to trap the checkpo...
- 6685 Views
- 3 replies
- 0 kudos
- 0 kudos
The problem is that on startup if a stream fails, it would never hit the awaitAnyTermination? I almost want to take that while loop and put it on a background thread to start that at the beginning and then fire all the streams afterward... not sure ...
- 0 kudos
- 6968 Views
- 3 replies
- 3 kudos
Resolved! Turn spark.sql query into scala function
Hello,I'm learning Scala / Spark and try to understand what's wrong with my function:I have a spark.sql query, stored in a variable:val uViewName = spark.sql(""" SELECT v.Data_View_Name FROM apoHierarchy AS h INNER JOIN apoView AS v ON h.View_N...
- 6968 Views
- 3 replies
- 3 kudos
- 3 kudos
try add .first()(0) it will return only value from first row/column as currently you are returning Dataset: var uViewName = spark.sql(s""" SELECT v.Data_View_Name FROM apoHierarchy AS h INNER JOIN apoView AS v ON h.View_Name = v.Context_View_N...
- 3 kudos
- 5176 Views
- 2 replies
- 1 kudos
Resolved! How to test Kafka connectivity from a Databricks notebook
My structured streaming job is failing as it's unable to connect to Kafka. I believe the issue is with Spark. How can I isolate if it's a Spark library issue or an actual network issue.
- 5176 Views
- 2 replies
- 1 kudos
- 1 kudos
The below code snippet can be used to test the connectivityimport java.util.Arrays import java.util.Properties import org.apache.kafka.clients.admin.AdminClient import org.apache.kafka.clients.admin.AdminClientConfig import org.apache.kafka.clients.a...
- 1 kudos
- 9680 Views
- 5 replies
- 5 kudos
Resolved! Using shared python wheels for job compute clusters
We have a GitHub workflow that generates a python wheel and uploads to a shared S3 available to our Databricks workspaces. When I install the Python wheel to a normal compute cluster using the path approach, it correctly installs the Python wheel and...
- 9680 Views
- 5 replies
- 5 kudos
- 5 kudos
You can mount S3 as a DBFS folder then set that library in "cluster" -> "libraries" tab -> "install new" -> "DBFS"
- 5 kudos
- 4080 Views
- 2 replies
- 5 kudos
Resolved! Different configurations for same Databricks Runtime version
Hi all,On my DBR installations, s3a scheme is mapped to shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem. On my customer's DBR installations it is mapped to com.databricks.s3a.S3AFileSystem.We both use the same DBR runtime, and none of us has...
- 4080 Views
- 2 replies
- 5 kudos
- 5 kudos
@Yoni Au​ , If both of you are using the same DBR version, then you should not find any difference. As @Hubert Dudek​ mentioned, there might be some spark configuration change made on one of the clusters. Also, it's worth checking for any cluster sco...
- 5 kudos
- 2788 Views
- 1 replies
- 2 kudos
I cannot access databricks community edition account
Last week, I cannot loginto https://community.cloud.databricks.com/login.html all of a sudden. I tried to set the password, also didn't receive the reset email. It says "Invalid email address or password Note: Emails/usernames are case-sensitive".I e...
- 2788 Views
- 1 replies
- 2 kudos
- 2 kudos
@Kaniz Fatma​ can help, additionally you can open ticket here https://help.databricks.com/s/contact-us
- 2 kudos
- 6495 Views
- 4 replies
- 8 kudos
Resolved! init_script error during cluster creation - 101: Network is unreachable
When I run the init_script during cluster creationapt-get update && apt-get install -y ffmpeg libsndfile1-devI get an error in cluster logs E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/o/openal-soft/libopenal1_1.19.1-1_amd64.deb ...
- 6495 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi @Sergii Ivakhno​ Could you please check if outbound TCP access for port 80 is allowed in the security group.
- 8 kudos
- 1093 Views
- 0 replies
- 0 kudos
PROCESS OF APPLYING FOR OMAN E-VISA The Oman e-Visa was initiated in 2018, for making the process easy. Presently, 220 countries in the world are elig...
PROCESS OF APPLYING FOR OMAN E-VISAThe Oman e-Visa was initiated in 2018, for making the process easy. Presently, 220 countries in the world are eligible to apply for Oman e-Visa. Tourists can apply for visas online by submitting the Oman visa applic...
- 1093 Views
- 0 replies
- 0 kudos
- 6861 Views
- 4 replies
- 4 kudos
Resolved! Cant find reason but suddenly new Jobs are getting huge job id numbers. example 945270539673815
Created Job ID is suddenly started to make huge numbers, and that is now making problems in Terraform plan, cause int is too big Error: strconv.ParseInt: parsing "945270539673815": value out of rangeIm new on the board and pretty new with Databricks ...
- 6861 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Jere Karhu​ , In case you are using the Job/Run id in API, please be advised that you will need to change the client-side logic to process int64/long and expect a random number. In some cases, you just need to change the declared type in their so...
- 4 kudos
- 6302 Views
- 3 replies
- 3 kudos
Resolved! Importing MongoDB with field names containing spaces
I am currently using a Python notebook with a defined schema to import fairly unstructured documents in MongoDB. Some of these documents have spaces in their field names. I define the schema for the MongoDB PySpark connector like the following:Struct...
- 6302 Views
- 3 replies
- 3 kudos
- 3 kudos
Solution: It turns out the issue is not the schema reading in, but the fact that I am writing to Delta tables, which do not currently support spaces. So, I need to transform them prior to dumping. I've been following a pattern of reading in raw data,...
- 3 kudos
- 4259 Views
- 1 replies
- 2 kudos
Resolved! Issue when reading .wav file
Hi, I am developing notebook to read .wav files and build Speech Matching Scenario. I have saved files in "/FileStore/tables/doors_and_corners_kid_thats_where_they_get_you.wav".When I wrote code like thisfrom scipy.io import wavfileimport numpy as np...
- 4259 Views
- 1 replies
- 2 kudos
- 2 kudos
Try to prefix it with dbfs dbfs:/FileStore or /dbfs/FileStore
- 2 kudos
- 9873 Views
- 1 replies
- 5 kudos
Resolved! How to Overwrite Using pyspark's JDBC without loosing constraints on table columns
Hello,My table has primary key constraint on a perticular column, Im loosing primary key constaint on that column each time I overwrite the table , What Can I do to preserve it? Any Heads up would be appreciatedTried Belowdf.write.option("truncate", ...
- 9873 Views
- 1 replies
- 5 kudos
- 5 kudos
@Abeeya .​ , Mode "truncate", is correct to preserve the table. However, when you want to add a new column (mismatched schema), it wants to drop it anyway.
- 5 kudos
- 2595 Views
- 1 replies
- 0 kudos
How to resolve Quickbooks error 12007
QuickBooks error 12007 occurs when an update time out occurs. QuickBooks may encounter this error when it cannot connect to the internet if it's unable to access the server. If you want to know its solutions then check out our latest blog on this.
- 2595 Views
- 1 replies
- 0 kudos
- 0 kudos
How to Resolve QuickBooks Payroll Update Error 12007?For various possible causes of the QB payroll update error 12007, you need to perform different troubleshooting procedures. Follow the solutions in their given sequence to fix this QuickBooks error...
- 0 kudos
- 10991 Views
- 11 replies
- 9 kudos
Resolved! Hi Experts i am new to data bricks and i want to know how data bricks supports real time reporting needs in Business intelligence?
Delta lake have 3 levels to maintain data quality ( bronze , silver and gold tables ) but this supports the reporting and BI solutions how does this supports the streaming analytics ?example : I have an app that loads all the operational data in adls...
- 10991 Views
- 11 replies
- 9 kudos
- 9 kudos
@Basavaraj Angadi​ , Why? For simplicity, cost savings etc.You can make it work with 2 'containers' but it is not necessary.
- 9 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
