- 3872 Views
- 2 replies
- 6 kudos
Resolved! Can anyone help me to understand one question in PracticeExam-DataEngineerAssociate?
It is the practice exam for data engineer associateThe question is:A data engineering team has created a series of tables using Parquet data stored in an external system. The team is noticing that after appending new rows to the data in the external ...
- 3872 Views
- 2 replies
- 6 kudos
- 6 kudos
Not an answer, just asking the databricks folks to clarify:I would also like to understand this. If there is no event emitted from the external parquet table (push) , and no active pulling or refreshing from the delta table side (pull), how is the un...
- 6 kudos
- 7438 Views
- 4 replies
- 5 kudos
Disable dbutils.fs.put() write to console "Wrote x bytes"
Hey all, does anyone know how to suppress the output of dbutils.fs.put() ?
- 7438 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Jordan Fox​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 5 kudos
- 4140 Views
- 5 replies
- 5 kudos
Databricks doesn't stop compute resources in GCP
I started using Databricks in Google Cloud but it charges some unexpected costs. When I create a cluster I notice some compute resources being created in GCP but when I stop the cluster these resources are still up and never shut down. This issue res...
- 4140 Views
- 5 replies
- 5 kudos
- 5 kudos
The answer to the question about the kubernetes cluster regardless of dbx compute and dwh resources running is provided in this thread: https://community.databricks.com/s/question/0D58Y00009TbWqtSAF/auto-termination-for-clusters-jobs-and-delta-live-t...
- 5 kudos
- 3947 Views
- 6 replies
- 2 kudos
Setting up my first DLT Pipeline with 3rd party JSON data
I'm getting an error when I try to create a DLT Pipeline from a bunch of third-party app-usage data we have. Here's the error message:Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema. Please upgrade your Delta table ...
- 3947 Views
- 6 replies
- 2 kudos
- 2 kudos
I found this other forum thread that looks potentially useful, but I can’t figure out either how to translate it to SQL to handle JSON, nor how to get the pipeline I’m working with to interpret the Python. When I switch to Python, it complains about ...
- 2 kudos
- 3614 Views
- 4 replies
- 0 kudos
What determines whether an experiment's "Rename/Permissions/Delete" context menu is active or grayed out in the Experiments page?
I have a couple of experiments in the machine learning workspace. Some I want to delete since they are clutter/were created just to test out the platform. However I can't because the options to delete them is grayed out (see pictures below)I was wond...
- 3614 Views
- 4 replies
- 0 kudos
- 0 kudos
It looks like from the documentation the user should have the 'can Manage' permission:https://docs.databricks.com/security/access-control/workspace-acl.html#mlflow-experiment-permissions-1
- 0 kudos
- 7595 Views
- 5 replies
- 4 kudos
Send custom logs to AWS cloudwatch from Notebook
I would like to send some custom logs (in Python) from my Databricks notebook to AWS Cloudwatch. For example: df = spark.read.json(".......................")logger.info("Successfully ingested data from json")Has someone succeeded in doing this before...
- 7595 Views
- 5 replies
- 4 kudos
- 4 kudos
Hi, You can integrate, please refer: https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/ and also you can configure audit logging to S3 and redirect it to cloudwatch from AWS. , refer: https://aws.amazon.com/blogs/mt/how...
- 4 kudos
- 1499 Views
- 1 replies
- 2 kudos
Resolved! " Please migrate to `databricks_group_role` "
With Databricks Terraform connector version 1.2.0, I use the following to make the AWS instance profile available to all users in the workspace:// Create AWS instance profileresource "aws_iam_instance_profile" "this" { name = "${var.prefix}_instance_...
- 1499 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello, I opened a support case a couple of months ago specifically about this. The answer I got was"Terraform team will revert the deprecated resource and update the document accordingly.", which has not happened so far. Either provide documentation ...
- 2 kudos
- 2059 Views
- 3 replies
- 0 kudos
Blank Page after Logging In
On Feb 8 Singapore time, our Singapore workspace displayed a blank page (no interface or content) after login. Meanwhile our workspace in Tokyo reason worked normally. This lasted whole day and none of our troubleshooting yielded any clues. Then ever...
- 2059 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Lawrence Chen​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 0 kudos
- 2950 Views
- 3 replies
- 4 kudos
Resolved! Data size inflates massively while ingesting
GoalImport and consolidate GBs / TBs of local data in 20-mb chunk parquet files into Databricks / Delta lake / partitioned tables.What I've DoneI took a small subset of data, roughly 72.5 GB and ingested using streaming below. The data is already seq...
- 2950 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Erik Louie​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 4 kudos
- 8746 Views
- 5 replies
- 2 kudos
Resolved! Errors on Python API for DeltaTables on Databricks Spark 10.4 LST
Hi,I am using an (Azure) Databricks Compute cluster in a Jupyter notebook using the Databricks connect Python package. Our cluster is on Databrick runtime 10.4 LST and coherently I am using the databricks-connect==10.4.18.In my notebook I am able to ...
- 8746 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Maarten van Raaij​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 2 kudos
- 1204 Views
- 1 replies
- 1 kudos
Resolved! Didn't receive my certification
I took my databrick certification exam on Feb-08-2023 however did'nt receive my certification untill now . Can i get help ?
- 1204 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Djoumoi Moinsalima​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 1 kudos
- 1728 Views
- 1 replies
- 0 kudos
Unable to connect to Azure Cosmos DB Cassandra API table using Azure databricks job
Getting below error:Query [id = , runId = ] terminated with exception: Failed to open native connection to Cassandra at {<name>.cassandra.cosmosdb.azure.com:10350} :: Method com/microsoft/azure/cosmosdb/cassandra/CosmosDbConnectionFactory$.createSess...
- 1728 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, The error looks like there is an issue connecting to csms-ws-ddicsg-dev-001.cassandra.cosmosdb.azure.com:10350. Could you please reverify this in networking config? Also, it will be helpful if you raise an Azure case simultaneously to check the n...
- 0 kudos
- 2575 Views
- 5 replies
- 1 kudos
Resolved! Data Engineer Certificate and badge not received
I have cleared the certification exam on 26th January 2023 with 91.11%, but still haven't received the certificate. I had given the exam with a different mail ID but I'm not receiving any emails from Databricks to that mail ID.​I've also raised a tic...
- 2575 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello team, I have answered via salesforce ticket and sent Naeem their sign on link and resent the badge/certificate email. Thanks!
- 1 kudos
- 6755 Views
- 8 replies
- 1 kudos
Mounting File Share in init script of cluster
we have a flow where we have to process chunk of files from file share. currently we are moving the files first to storage account and then post processing move files back to file share again. this is adding to the execution time for moving files bac...
- 6755 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @Rahul Samant is this issue solved Please help we are also facing same issues
- 1 kudos
- 4946 Views
- 6 replies
- 10 kudos
Disable dbutils suggestion
I would like to turn off or suppress this message which is returned from the dbutils library. %r files <- dbutils.fs.ls("/dbfs/tmp/") For prettier results from dbutils.fs.ls(<dir>), please use `%fs ls <dir>`How can I do this?
- 4946 Views
- 6 replies
- 10 kudos
- 10 kudos
Hi @James Smith​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 10 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
