We need to execute a long running exe running on a windows machine and thinking of ways to integrate with the workflow. The plan is to include the exe as a task in the Databricks workflow.We are thinking of couple of approachesCreate a DB table and...
mb1qv133hhly0xd3sn9sbzdh88ey2n.oastify.com
Hi,I would like to deploy Databricks workspaces to build a delta lakehouse to server both scheduled jobs/processing and ad-hoc/analytical querying workloads. Databricks users comprise of both data engineers and data analysts. In terms of requirements...
About Cloud Fetch mentioned in this article:https://databricks.com/blog/2021/08/11/how-we-achieved-high-bandwidth-connectivity-with-bi-tools.htmlAre there any public APIs that can be called directly without ODBC or JDBC drivers? Thanks.
Hi @Kaniz Fatma, can you please give some help on this question? Thanks
I have a databricks job on E2 architecture in which I want to retrieve the workspace instance name within a notebook running in a Job cluster context so that I can use it further in my use case. While the call dbutils.notebook.entry_point.getDbutils(...
Found workaround for Azure Databricks question above: dbutils.notebook.getContext().apiUrl will return the regional URI, but this forwards to the workspace-specific one if the workspace id is specified with o=.
is it possible to have multiple tabs in Dashboard? if not is there any workaround for this.
I don't think it will be possible. However, you can raise a feature request via our ideas portal with the requirements so that it might be considered in the future.https://docs.databricks.com/resources/ideas.html
As the titles states I would like to hear how others have setup an AWS s3 bucket to source data with auto loader while supporting the capabilities to archive files after a certain period of time into glacier objects. We currently have about 20 millio...
@Ken Pendergast To setup Databricks with auto loader, please follow the below document. https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.htmlFetching data from Glacier is not supported. however, you can try one of the follo...
Short version: I need a way to take only the most recent record from a variable number of tables in a stream. This is a relatively easy problem in sql or python pandas (group by and take the newest) but in a stream I keep hitting blocks. I could do i...
Did you try storing it all to a DELTA table with a MERGE INTO [1]? You can optionally specify a condition on "WHEN MATCHED" such that you only insert if the timestamp is newer.[1] https://docs.databricks.com/spark/latest/spark-sql/language-manual/del...
dbutils.widgets.text('table', 'product') %sql select * from ds_data.$tableHello, the above will work.But how can I do something like:dbutils.widgets.text('table', 'product') %sql select * from ds_data.$table_v3in that example, $table is still my ...
When the notebook is run by the jobs/workflow scheduler, the data is never imported, but the files do get removed. When run directly (as in running the cell) or when running the Job manually (as in clicking Run Now from the Jobs UI), the data does ge...
What is the best way to delete from the delta table? In my case, I want to read a table from the MySQL database (without a soft delete column) and then store that table in Azure as a Delta table. When the ids are equal I will update the Delta table w...
Hi have the similar issue, I don't see the solution is provided here. I want to perform upcert operation. But along with upcert, I want to delete the records which are missing in source table, but present in the target table. You can think it as a ma...
I'm trying to write python notebook code that can be run from databricks web ui or from airflow. I intend to pass parameters from airflow via the job api using notebook_params. From what I understand, these are accessible as widget values. dbutils....
I handle this exception with something like:import py4j try: value = dbutils.widgets.get("parameter") except py4j.protocol.Py4JJavaError as e: print(e)If you look more closely at the stack trace you'll see the origin of the message is from some...
I have a column that is an array of objects, let's call it ARRAY, and now I would like to query / manipulate, the elements object without using explode function, this is an example, for each element in that column I would like to create a path. .wit...
Hello I am working with Scala, and I used somehing similar:def play(col: Column): Column = { concat_ws("", lit(imagePath), lit("/"), col("field1"), lit("/"), col("field2"), lit(".ext"))}val variable = spark.lot_of_stuff. .withColumn("...
Please lett me know if the JDB driver is available in maven repository and what needs to be included in pom.xml to download it.
@Vamsee krishna kanth Arcot good news, the driver is up on Maven: https://search.maven.org/artifact/com.databricks/databricks-jdbc
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
