- 14077 Views
- 7 replies
- 2 kudos
Schema evolution issue
Hi AllI am loading some data using auto loader but am having trouble with Schema evolution.A new column has been added to the data I am loading and I am getting the following error:StreamingQueryException: Encountered unknown field(s) during parsing:...
- 14077 Views
- 7 replies
- 2 kudos
- 2 kudos
I agree that hints are the way to go if you have the schema available but the whole point of schema evolution is that you might not always know the schema in advance.I received a similar error with a similar streaming query configuration. The issue w...
- 2 kudos
- 3920 Views
- 2 replies
- 3 kudos
Resolved! Generic user account and personal access token to Azure Datarbicks
Is there a way to create a generic user account and personal access token to connect to databricks. I have Azure build pipeline and VSCode test that is using my personal access token for running builds and tests.
- 3920 Views
- 2 replies
- 3 kudos
- 3 kudos
You can create a service account (principle) for jobs, applications etc. Here's a link to the docs:https://docs.databricks.com/administration-guide/users-groups/service-principals.html
- 3 kudos
- 4447 Views
- 4 replies
- 2 kudos
Why set up audit log delivery in databricks GCP fails ?
I am trying to set up audit log delivery in google cloud. I have followed this page https://docs.gcp.databricks.com/administration-guide/account-settings-gcp/log-delivery.html and have added log-delivery@databricks-prod-master.iam.gserviceaccount.co...
- 4447 Views
- 4 replies
- 2 kudos
- 2 kudos
I would suggest, contacting your Databricks accounts representative for this. They would be able to check if something went wrong with your workspace subscription.
- 2 kudos
- 2918 Views
- 2 replies
- 2 kudos
How do I process each new record when using autoloader.
For instance, I'm ingesting webhook data into a delta table with autoloader and need to run a process for each new record as it arrives.
- 2918 Views
- 2 replies
- 2 kudos
- 2 kudos
With autoloader, you can do something like changelog and record data about operations performed on each micro batch - like affected id, I/U/D, timestamp etc..Then you can make use of this changelog table, and run subsequent processes for each row aff...
- 2 kudos
- 3116 Views
- 2 replies
- 3 kudos
AWS OR AZURE OR GCLOUD??
I want to know whether which cloud is better to learn and which cloud services has more career opportunities.
- 3116 Views
- 2 replies
- 3 kudos
- 3 kudos
As addition to @Kaniz Fatma​ great comparison article, cloud skills are generally transferable across other providers. It is the same concept just with different names (eg: EC2 / Azure VM / Google Compute Engine). Learning cloud in general is a good ...
- 3 kudos
- 5646 Views
- 4 replies
- 3 kudos
Resolved! "SparkSecurityException: Cannot read sensitive key" error when reading key from Spark config
In Databricks 10.1 it is possible to define in the "Spark Config" of the cluster something like:spark.fernet {{secrets/myscope/encryption-key}} . In my case my scopes are tied to Azure Key Vault.With that I can make a query as follows:%sql SELECT d...
- 5646 Views
- 4 replies
- 3 kudos
- 3 kudos
This solution exposes the entire secret if I use commands like belowsql("""explain select upper("${spark.fernet.email}") as data """).display()Please dont use this
- 3 kudos
- 2619 Views
- 3 replies
- 2 kudos
Firefox only - copying from notebook table output copies cell contents instead
Firefox only - copying from notebook table output copies cell contents instead in Firefox (and firefox based browsers)
- 2619 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Jim Kutter​, I have gone ahead and put in a ticket for you regarding this. Your Databricks representative will be in touch with you regarding the status. Thank you for your patience!
- 2 kudos
- 1498 Views
- 1 replies
- 0 kudos
What is the instrument of activity of Go?
ORDER NOW >>> https://www.outlookindia.com/outlook-spotlight/go-reviews-is-go-legit-or-news-207720 Go"Go" may upgrade mental state many days. They might work on mental concentration in half a month. This regular item might help focus levels and make ...
- 1498 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kaniz Fatma​ - spam post
- 0 kudos
- 60192 Views
- 24 replies
- 4 kudos
Resolved! Extracting data from a multi-layered JSON object
I have a table in databricks called owner_final_delta with a column called contacts that holds data with this structure:array<struct<address:struct<apartment:string,city:string,house:string,poBox:string,sources:array<string>,state:string,street:strin...
- 60192 Views
- 24 replies
- 4 kudos
- 4 kudos
Have you tried to use the explode function for that column with the array?df.select(explode(df.emailId).alias("email")).show()----------Also, if you are a SQL lover, you can instead use the Databricks syntax for querying a JSON seen here.
- 4 kudos
- 4617 Views
- 1 replies
- 0 kudos
How to add unique consecutive id to delta lake table
In Databricks I have a existing delta table, In which i want to add one more column, as Id so that each row has unique id no and It is consecutive (how primary key is present in sql).So far I have tried converting delta table to pyspark dataframe and...
- 4617 Views
- 1 replies
- 0 kudos
- 0 kudos
How about defining an identity column as below?GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( [ START WITH start ] [ INCREMENT BY step ] ) ]https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html#parameters
- 0 kudos
- 3085 Views
- 2 replies
- 1 kudos
Resolved! Add an Instance Profile to a DLT job cluster
@Tomasz Bacewicz​ I've got another, related question for you about the job cluster that is spun up for DLT jobs. Adding the JSON strings for our required E2 tags worked like a charm, but now I need to attach an existing Instance Profile since I'm tr...
- 3085 Views
- 2 replies
- 1 kudos
- 1 kudos
@Brad Sheridan​ To do that you have to add the aws_attributes tag within a cluster configuration and there you have the ability to add an instance_profile_arn like so:"clusters": [ { "label": "default", "aws_attributes": { ...
- 1 kudos
- 9062 Views
- 0 replies
- 1 kudos
We are seeing "BOOTSTRAP_TIMEOUT" issue in a new workspace.
When attempting to deploy/start an Databricks cluster on AWS through the UI, the following error consistently occurs:Bootstrap Timeout:[id: InstanceId(i-093caac78cdbfa7e1), status: INSTANCE_INITIALIZING, workerEnvId:WorkerEnvId(workerenv-335698072713...
- 9062 Views
- 0 replies
- 1 kudos
- 15216 Views
- 8 replies
- 12 kudos
Resolved! How to programmatically create a widget?
I know the dbutils.widget.text to create a widget in a notebook. So for me the workflow 1. Having a notebook2. Use dbutils.widget.text (or other type of widgets) once in a notebook cell to create a widget3. Remove the cell containing dbutils.widget...
- 15216 Views
- 8 replies
- 12 kudos
- 12 kudos
@Mehdi BEN ABDESSELEM​ , Steps for Creating a Basic WidgetStep 1: Create a New ProjectTo create a new project in Android Studio, please refer to How to Create/Start a New Project in Android Studio. We are implementing it for both Java and Kotlin lang...
- 12 kudos
- 6562 Views
- 3 replies
- 6 kudos
Resolved! QuantileDiscretizer not respecting NumBuckets
I have set numBuckets and numBucketsArray for a group of columns to bin them into 5 buckets.Unfortunately the number of buckets does not seem to be respected across all columns even though there is variation within them.I have tried setting the relat...
- 6562 Views
- 3 replies
- 6 kudos
- 6 kudos
Thank you.What I did was:Apply QuntileBucketizer to Non-Zeros and specified a very small value (bottom 1%) to capture the lower bucket including zeroes.That fixed the issue! You can define your own splits which would work as well but the splits thems...
- 6 kudos
- 10480 Views
- 9 replies
- 6 kudos
Resolved! The Databricks-academy question
I'm learning the Data Engineeing with Databricks of Course, I have a question.if I run cmd4, it tells me an error.Course URL:https://customer-academy.databricks.com/learn/course/62/play/4290/providing-options-for-external-sources;lp=10Chapter: DE 4....
- 10480 Views
- 9 replies
- 6 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
