cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Nick_Hughes
by New Contributor III
  • 4410 Views
  • 9 replies
  • 4 kudos

Resolved! Does Databricks persist the alert results history?

Hi, this seems like a really basic feature to have, an alert is generated and spits out the email, but the URL doesn't take you to the events list that happened at that time but just the query in the editor (via the alert config screen). We're really...

  • 4410 Views
  • 9 replies
  • 4 kudos
Latest Reply
Prabakar
Databricks Employee
  • 4 kudos

@Nick Hughes​ I was looking at our ideas portal and could see an API feature was requested for the Alerts (DB-I-4289). To create an API for the alerts and pull data, we might need to keep the alert history persistent. So this feature should suffice y...

  • 4 kudos
8 More Replies
Sandeep
by Contributor III
  • 557 Views
  • 0 replies
  • 4 kudos

spark.apache.org

Per API docs on StreamingQuery.stop(), https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/streaming/StreamingQuery.htmlIt says, this stops the execution of this query if it is running and waits until the termination of the query exec...

  • 557 Views
  • 0 replies
  • 4 kudos
dimoobraznii
by New Contributor III
  • 6094 Views
  • 5 replies
  • 6 kudos

Resolved! Autoloader failed

I used autoloader with TriggerOnce = true and ran it for weeks with schedule. Today it broke:The metadata file in the streaming source checkpoint directory is missing. This metadatafile contains important default options for the stream, so the stream...

  • 6094 Views
  • 5 replies
  • 6 kudos
Latest Reply
Deepak_Bhutada
Contributor III
  • 6 kudos

Hi dimoobraznii (Customer)This error comes in streaming when someone makes changes to the streaming checkpoint directory manually or points some streaming type to the checkpoint of some other streaming type. Please check if any changes were made to t...

  • 6 kudos
4 More Replies
Anonymous
by Not applicable
  • 2513 Views
  • 3 replies
  • 7 kudos

Resolved! How does 73% of the data go unused for analytics or decision-making?

Is Lakehouse the answer? Here's a good resource that was just published: https://dbricks.co/3q3471X

  • 2513 Views
  • 3 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

@Alexis Lopez​ - If @Dan Zafar​ 's or @Harikrishnan Kunhumveettil​'s answers solved the issue, would you be happy to mark one of their answers as best so other members can find the solution more easily?

  • 7 kudos
2 More Replies
Gapy
by New Contributor II
  • 1408 Views
  • 1 replies
  • 1 kudos

Auto Loader Schema-Inference and Evolution for parquet files

Dear all,will (and when) will Auto Loader also support Schema-Inference and Evolution for parquet files, at this point it is only for JSON and CSV supported if i am not mistaken?Thanks and regards,Gapy

  • 1408 Views
  • 1 replies
  • 1 kudos
Latest Reply
Sandeep
Contributor III
  • 1 kudos

@Gasper Zerak​ , This will be available in near future (DBR 10.3 or later). Unfortunately, we don't have an SLA at this moment.

  • 1 kudos
Maverick1
by Valued Contributor II
  • 4096 Views
  • 10 replies
  • 9 kudos

Resolved! Lineage between model and source code breaks on movement of source notebook. How to rectify it?

If there is a registered model and it is linked with a notebook, then the lineage breaks if you move the notebook to a different path or even pull/upload a new version of the notebook.This is not good because when someone doing its development/testin...

  • 4096 Views
  • 10 replies
  • 9 kudos
Latest Reply
sean_owen
Databricks Employee
  • 9 kudos

I also cannot reproduce this, with these exact steps (I think). After moving the notebook and moving it back, the link to it (and link to the revision) still works as expected. You are using MLflow built in to Databricks right?

  • 9 kudos
9 More Replies
RantoB
by Valued Contributor
  • 6925 Views
  • 1 replies
  • 3 kudos

Resolved! %run file not found

Hi,I was using the following command to import variables and functions from an other notebook :%run ./utilsFor some reason it is not working any more and gives me this message :Exception: File `'./utils.py'` not found.utils.py is still at the same pl...

  • 6925 Views
  • 1 replies
  • 3 kudos
Latest Reply
RantoB
Valued Contributor
  • 3 kudos

Finally I just solved my issue.Actually, in the same cell I wrote a comment starting with # and it was not working because of that...

  • 3 kudos
bdc
by New Contributor III
  • 9336 Views
  • 4 replies
  • 3 kudos

Is it possible to access a variable in markdown cells?

I saw a similar question in a discussion 5 years ago and back then this option was not available. https://community.databricks.com/s/question/0D53f00001HKHhNCAX/markup-in-databricks-notebookI wonder if this feature has been added. It is possible to d...

  • 9336 Views
  • 4 replies
  • 3 kudos
Latest Reply
bdc
New Contributor III
  • 3 kudos

When I click on the feedback link under workspace/help, it opens a page with page not found.

  • 3 kudos
3 More Replies
FemiAnthony
by New Contributor III
  • 3835 Views
  • 4 replies
  • 3 kudos

Resolved! Location of customer_t1 dataset

Can anyone tell me how I can access the customer_t1 dataset that is referenced in the book "Delta Lake - The Definitive Guide "? I am trying to follow along with one of the examples.

  • 3835 Views
  • 4 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Some files are visualized here https://github.com/vinijaiswal/delta_time_travel/blob/main/Delta%20Time%20Travel.ipynb but it is quite strange that there is no source in repository. I think only one way is to write to Vini Jaiswal on github.

  • 3 kudos
3 More Replies
Mohit_m
by Valued Contributor II
  • 788 Views
  • 1 replies
  • 2 kudos

docs.databricks.com

Are EBS volumes used by Databricks Clusters are encrypted especially the root volumes

  • 788 Views
  • 1 replies
  • 2 kudos
Latest Reply
Mohit_m
Valued Contributor II
  • 2 kudos

Yes these EBS volumes are encrypted. Earlier root volume encryptions were not supported but recently this encryption is also enabled (since Apr, 2021)please find more details on the below docs pagehttps://docs.databricks.com/clusters/configure.html#e...

  • 2 kudos
FemiAnthony
by New Contributor III
  • 4954 Views
  • 6 replies
  • 5 kudos

Resolved! /dbfs is empty

Why does /dbfs seem to be empty in my Databricks cluster ?If I run %sh ls /dbfsI get no output.I am looking for the databricks-datasets subdirectory ? I can't find it under /dbfs

  • 4954 Views
  • 6 replies
  • 5 kudos
Latest Reply
FemiAnthony
New Contributor III
  • 5 kudos

Thanks @Prabakar Ammeappin​ 

  • 5 kudos
5 More Replies
Sandesh87
by New Contributor III
  • 1836 Views
  • 3 replies
  • 2 kudos

Resolved! log error to cosmos db

Objective:- Retrieve objects from an S3 bucket using a 'get' api call, write the retrieved object to azure datalake and in case of errors like 404s (object not found) write the error message to cosmos DB"my_dataframe" consists of the a column (s3Obje...

  • 1836 Views
  • 3 replies
  • 2 kudos
Latest Reply
User16763506477
Contributor III
  • 2 kudos

Hi @Sandesh Puligundla​  issue is that you are using spark context inside foreachpartition. You can create a dataframe only on the spark driver. Few stack overflow references https://stackoverflow.com/questions/46964250/nullpointerexception-creatin...

  • 2 kudos
2 More Replies
UM
by New Contributor II
  • 2353 Views
  • 2 replies
  • 4 kudos

Resolved! Identifying the right tools for the job

Hi all, thank you for taking the time to attend to my post. A background to preface, my team and I have been prototyping an ML model that we would like to push into the production and deployment phase. We have been prototyping on Jupyter Notebooks bu...

untitled
  • 2353 Views
  • 2 replies
  • 4 kudos
Latest Reply
Dan_Z
Databricks Employee
  • 4 kudos

For production model serving, why not just use MLflow Model Serving? You just code it up/import it with the notebooks, then Log it using MLflow, then Register it with the MLflow Registry, then there is a nice UI to serve it using Model Serving. It wi...

  • 4 kudos
1 More Replies
Mohit_m
by Valued Contributor II
  • 1146 Views
  • 1 replies
  • 4 kudos

Enabling of Task Orchestration feature in Jobs via API as well Databricks supports the ability to orchestrate multiple tasks within a job. You must en...

Enabling of Task Orchestration feature in Jobs via API as wellDatabricks supports the ability to orchestrate multiple tasks within a job. You must enable this feature in the admin console. Once enabled, this feature cannot be disabled. To enable orch...

  • 1146 Views
  • 1 replies
  • 4 kudos
Latest Reply
Prabakar
Databricks Employee
  • 4 kudos

@Mohit Miglani​ this will be really helpful for those who prefer CLI / API more than the UI.

  • 4 kudos
SEOCO
by New Contributor II
  • 2733 Views
  • 3 replies
  • 3 kudos

Passing parameters from DevOps Pipeline/release to DataBricks Notebook

Hi,This is all a bit new to me.Does anybody have any idea how to pass a parameter to the Databricks notebook.I have a DevOps pipeline/release that moves my databricks notebooks towards QA and Production environment. The only problem I am facing is th...

  • 2733 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Mario Walle​ - If @Hubert Dudek​'s answer solved the issue, would you be happy to mark his answer as best so that it will be more visible to other members?

  • 3 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels