- 4193 Views
- 4 replies
- 2 kudos
Why set up audit log delivery in databricks GCP fails ?
I am trying to set up audit log delivery in google cloud. I have followed this page https://docs.gcp.databricks.com/administration-guide/account-settings-gcp/log-delivery.html and have added log-delivery@databricks-prod-master.iam.gserviceaccount.co...
- 4193 Views
- 4 replies
- 2 kudos
- 2 kudos
I would suggest, contacting your Databricks accounts representative for this. They would be able to check if something went wrong with your workspace subscription.
- 2 kudos
- 2755 Views
- 2 replies
- 2 kudos
How do I process each new record when using autoloader.
For instance, I'm ingesting webhook data into a delta table with autoloader and need to run a process for each new record as it arrives.
- 2755 Views
- 2 replies
- 2 kudos
- 2 kudos
With autoloader, you can do something like changelog and record data about operations performed on each micro batch - like affected id, I/U/D, timestamp etc..Then you can make use of this changelog table, and run subsequent processes for each row aff...
- 2 kudos
- 2970 Views
- 2 replies
- 3 kudos
AWS OR AZURE OR GCLOUD??
I want to know whether which cloud is better to learn and which cloud services has more career opportunities.
- 2970 Views
- 2 replies
- 3 kudos
- 3 kudos
As addition to @Kaniz Fatma​ great comparison article, cloud skills are generally transferable across other providers. It is the same concept just with different names (eg: EC2 / Azure VM / Google Compute Engine). Learning cloud in general is a good ...
- 3 kudos
- 5471 Views
- 4 replies
- 3 kudos
Resolved! "SparkSecurityException: Cannot read sensitive key" error when reading key from Spark config
In Databricks 10.1 it is possible to define in the "Spark Config" of the cluster something like:spark.fernet {{secrets/myscope/encryption-key}} . In my case my scopes are tied to Azure Key Vault.With that I can make a query as follows:%sql SELECT d...
- 5471 Views
- 4 replies
- 3 kudos
- 3 kudos
This solution exposes the entire secret if I use commands like belowsql("""explain select upper("${spark.fernet.email}") as data """).display()Please dont use this
- 3 kudos
- 2482 Views
- 3 replies
- 2 kudos
Firefox only - copying from notebook table output copies cell contents instead
Firefox only - copying from notebook table output copies cell contents instead in Firefox (and firefox based browsers)
- 2482 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Jim Kutter​, I have gone ahead and put in a ticket for you regarding this. Your Databricks representative will be in touch with you regarding the status. Thank you for your patience!
- 2 kudos
- 1384 Views
- 1 replies
- 0 kudos
What is the instrument of activity of Go?
ORDER NOW >>> https://www.outlookindia.com/outlook-spotlight/go-reviews-is-go-legit-or-news-207720 Go"Go" may upgrade mental state many days. They might work on mental concentration in half a month. This regular item might help focus levels and make ...
- 1384 Views
- 1 replies
- 0 kudos
- 0 kudos
@Kaniz Fatma​ - spam post
- 0 kudos
- 57922 Views
- 24 replies
- 4 kudos
Resolved! Extracting data from a multi-layered JSON object
I have a table in databricks called owner_final_delta with a column called contacts that holds data with this structure:array<struct<address:struct<apartment:string,city:string,house:string,poBox:string,sources:array<string>,state:string,street:strin...
- 57922 Views
- 24 replies
- 4 kudos
- 4 kudos
Have you tried to use the explode function for that column with the array?df.select(explode(df.emailId).alias("email")).show()----------Also, if you are a SQL lover, you can instead use the Databricks syntax for querying a JSON seen here.
- 4 kudos
- 4459 Views
- 1 replies
- 0 kudos
How to add unique consecutive id to delta lake table
In Databricks I have a existing delta table, In which i want to add one more column, as Id so that each row has unique id no and It is consecutive (how primary key is present in sql).So far I have tried converting delta table to pyspark dataframe and...
- 4459 Views
- 1 replies
- 0 kudos
- 0 kudos
How about defining an identity column as below?GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( [ START WITH start ] [ INCREMENT BY step ] ) ]https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-table-using.html#parameters
- 0 kudos
- 2906 Views
- 2 replies
- 1 kudos
Resolved! Add an Instance Profile to a DLT job cluster
@Tomasz Bacewicz​ I've got another, related question for you about the job cluster that is spun up for DLT jobs. Adding the JSON strings for our required E2 tags worked like a charm, but now I need to attach an existing Instance Profile since I'm tr...
- 2906 Views
- 2 replies
- 1 kudos
- 1 kudos
@Brad Sheridan​ To do that you have to add the aws_attributes tag within a cluster configuration and there you have the ability to add an instance_profile_arn like so:"clusters": [ { "label": "default", "aws_attributes": { ...
- 1 kudos
- 8984 Views
- 0 replies
- 1 kudos
We are seeing "BOOTSTRAP_TIMEOUT" issue in a new workspace.
When attempting to deploy/start an Databricks cluster on AWS through the UI, the following error consistently occurs:Bootstrap Timeout:[id: InstanceId(i-093caac78cdbfa7e1), status: INSTANCE_INITIALIZING, workerEnvId:WorkerEnvId(workerenv-335698072713...
- 8984 Views
- 0 replies
- 1 kudos
- 14767 Views
- 8 replies
- 12 kudos
Resolved! How to programmatically create a widget?
I know the dbutils.widget.text to create a widget in a notebook. So for me the workflow 1. Having a notebook2. Use dbutils.widget.text (or other type of widgets) once in a notebook cell to create a widget3. Remove the cell containing dbutils.widget...
- 14767 Views
- 8 replies
- 12 kudos
- 12 kudos
@Mehdi BEN ABDESSELEM​ , Steps for Creating a Basic WidgetStep 1: Create a New ProjectTo create a new project in Android Studio, please refer to How to Create/Start a New Project in Android Studio. We are implementing it for both Java and Kotlin lang...
- 12 kudos
- 6350 Views
- 3 replies
- 6 kudos
Resolved! QuantileDiscretizer not respecting NumBuckets
I have set numBuckets and numBucketsArray for a group of columns to bin them into 5 buckets.Unfortunately the number of buckets does not seem to be respected across all columns even though there is variation within them.I have tried setting the relat...
- 6350 Views
- 3 replies
- 6 kudos
- 6 kudos
Thank you.What I did was:Apply QuntileBucketizer to Non-Zeros and specified a very small value (bottom 1%) to capture the lower bucket including zeroes.That fixed the issue! You can define your own splits which would work as well but the splits thems...
- 6 kudos
- 9985 Views
- 9 replies
- 6 kudos
Resolved! The Databricks-academy question
I'm learning the Data Engineeing with Databricks of Course, I have a question.if I run cmd4, it tells me an error.Course URL:https://customer-academy.databricks.com/learn/course/62/play/4290/providing-options-for-external-sources;lp=10Chapter: DE 4....
- 9985 Views
- 9 replies
- 6 kudos
- 4788 Views
- 6 replies
- 7 kudos
Where is Alerts in the sidebar?
Hi everyone,I can't seem to find Alerts in the sidebar, also my data-explorer looks different from what I see in the videos. Do I need to upgrade my environment? Thanks,K
- 4788 Views
- 6 replies
- 7 kudos
- 7 kudos
Hi group,After speaking with my rep, it appears that Databricks ALERTS is only for premium members even though that is not what is advertised on the site or in the documentation. This is unfortunate as data-quality is a concern for us and we don't fe...
- 7 kudos
- 16304 Views
- 8 replies
- 6 kudos
Resolved! logging.basicConfig not creating a file in Databricks
Hi,I am using the logger to log some parameters in my code and I want to save the file under DBFS. But for some reason the file is not getting created under DBFS. If I clear the state of the notebook and check the DBFS dir then file is present. Pleas...
- 16304 Views
- 8 replies
- 6 kudos
- 6 kudos
Perhaps PyCharm sets a different working directory, meaning the file ends up in another place. Try providing a full path.
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 579 | |
| 349 | |
| 287 |
